{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b518b04cbfe0"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "906e07f6e562"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c359f002e834"
      },
      "source": [
        "# Training Keras models with TensorFlow Cloud"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f5c893a15fac"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/guide/keras/training_keras_models_on_cloud\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/snapshot-keras/site/en/guide/keras/training_keras_models_on_cloud.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/keras-team/keras-io/blob/master/guides/training_keras_models_on_cloud.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/guide/keras/training_keras_models_on_cloud.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b1c0246f8536"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "[TensorFlow Cloud](https://github.com/tensorflow/cloud) is a Python package that\n",
        "provides APIs for a seamless transition from local debugging to distributed training\n",
        "in Google Cloud. It simplifies the process of training TensorFlow models on the\n",
        "cloud into a single, simple function call, requiring minimal setup and no changes\n",
        "to your model. TensorFlow Cloud handles cloud-specific tasks such as creating VM\n",
        "instances and distribution strategies for your models automatically. This guide\n",
        "will demonstrate how to interface with Google Cloud through TensorFlow Cloud,\n",
        "and the wide range of functionality provided within TensorFlow Cloud. We'll start\n",
        "with the simplest use-case."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e015c75faba2"
      },
      "source": [
        "## Setup\n",
        "\n",
        "We'll get started by installing TensorFlow Cloud, and importing the packages we\n",
        "will need in this guide."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "99e5bc5e0ab8"
      },
      "outputs": [],
      "source": [
        "!pip install -q tensorflow_cloud"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "26113effabca"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_cloud as tfc\n",
        "\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e8568395c87b"
      },
      "source": [
        "## API overview: a first end-to-end example\n",
        "\n",
        "Let's begin with a Keras model training script, such as the following CNN:\n",
        "\n",
        "```python\n",
        "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
        "\n",
        "model = keras.Sequential(\n",
        "    [\n",
        "        keras.Input(shape=(28, 28)),\n",
        "        # Use a Rescaling layer to make sure input values are in the [0, 1] range.\n",
        "        layers.experimental.preprocessing.Rescaling(1.0 / 255),\n",
        "        # The original images have shape (28, 28), so we reshape them to (28, 28, 1)\n",
        "        layers.Reshape(target_shape=(28, 28, 1)),\n",
        "        # Follow-up with a classic small convnet\n",
        "        layers.Conv2D(32, 3, activation=\"relu\"),\n",
        "        layers.MaxPooling2D(2),\n",
        "        layers.Conv2D(32, 3, activation=\"relu\"),\n",
        "        layers.MaxPooling2D(2),\n",
        "        layers.Conv2D(32, 3, activation=\"relu\"),\n",
        "        layers.Flatten(),\n",
        "        layers.Dense(128, activation=\"relu\"),\n",
        "        layers.Dense(10),\n",
        "    ]\n",
        ")\n",
        "\n",
        "model.compile(\n",
        "    optimizer=keras.optimizers.Adam(),\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "    metrics=keras.metrics.SparseCategoricalAccuracy(),\n",
        ")\n",
        "\n",
        "model.fit(x_train, y_train, epochs=20, batch_size=128, validation_split=0.1)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "514f51a9a45d"
      },
      "source": [
        "To train this model on Google Cloud we just need to add a call to `run()` at\n",
        "the beginning of the script, before the imports:\n",
        "```python\n",
        "tfc.run()\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6e38288bb617"
      },
      "source": [
        "You don't need to worry about cloud-specific tasks such as creating VM instances\n",
        "and distribution strategies when using TensorFlow Cloud.\n",
        "The API includes intelligent defaults for all the parameters -- everything is\n",
        "configurable, but many models can rely on these defaults.\n",
        "\n",
        "Upon calling `run()`, TensorFlow Cloud will:\n",
        "\n",
        "- Make your Python script or notebook distribution-ready.\n",
        "- Convert it into a Docker image with required dependencies.\n",
        "- Run the training job on a GCP GPU-powered VM.\n",
        "- Stream relevant logs and job information.\n",
        "\n",
        "The default VM configuration is 1 chief and 0 workers with 8 CPU cores and\n",
        "1 Tesla T4 GPU."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3ab860e037c9"
      },
      "source": [
        "## Google Cloud configuration\n",
        "\n",
        "In order to facilitate the proper pathways for Cloud training, you will need to\n",
        "do some first-time setup. If you're a new Google Cloud user, there are a few\n",
        "preliminary steps you will need to take:\n",
        "\n",
        "1. Create a GCP Project;\n",
        "2. Enable AI Platform Services;\n",
        "3. Create a Service Account;\n",
        "4. Download an authorization key;\n",
        "5. Create a Cloud Storage bucket.\n",
        "\n",
        "Detailed first-time setup instructions can be found in the\n",
        "[TensorFlow Cloud README](https://github.com/tensorflow/cloud#setup-instructions),\n",
        "and an additional setup example is shown on the\n",
        "[TensorFlow Blog](https://blog.tensorflow.org/2020/08/train-your-tensorflow-model-on-google.html).\n",
        "\n",
        "## Common workflows and Cloud storage\n",
        "\n",
        "In most cases, you'll want to retrieve your model after training on Google Cloud.\n",
        "For this, it's crucial to redirect saving and loading to Cloud Storage while\n",
        "training remotely. We can direct TensorFlow Cloud to our Cloud Storage bucket for\n",
        "a variety of tasks. The storage bucket can be used to save and load large training\n",
        "datasets, store callback logs or model weights, and save trained model files.\n",
        "To begin, let's configure `fit()` to save the model to a Cloud Storage, and set\n",
        "up TensorBoard monitoring to track training progress."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "af5077731187"
      },
      "outputs": [],
      "source": [
        "def create_model():\n",
        "    model = keras.Sequential(\n",
        "        [\n",
        "            keras.Input(shape=(28, 28)),\n",
        "            layers.experimental.preprocessing.Rescaling(1.0 / 255),\n",
        "            layers.Reshape(target_shape=(28, 28, 1)),\n",
        "            layers.Conv2D(32, 3, activation=\"relu\"),\n",
        "            layers.MaxPooling2D(2),\n",
        "            layers.Conv2D(32, 3, activation=\"relu\"),\n",
        "            layers.MaxPooling2D(2),\n",
        "            layers.Conv2D(32, 3, activation=\"relu\"),\n",
        "            layers.Flatten(),\n",
        "            layers.Dense(128, activation=\"relu\"),\n",
        "            layers.Dense(10),\n",
        "        ]\n",
        "    )\n",
        "\n",
        "    model.compile(\n",
        "        optimizer=keras.optimizers.Adam(),\n",
        "        loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "        metrics=keras.metrics.SparseCategoricalAccuracy(),\n",
        "    )\n",
        "    return model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5f2e65d8f3a6"
      },
      "source": [
        "Let's save the TensorBoard logs and model checkpoints generated during training\n",
        "in our cloud storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fdc4f951281c"
      },
      "outputs": [],
      "source": [
        "import datetime\n",
        "import os\n",
        "\n",
        "# Note: Please change the gcp_bucket to your bucket name.\n",
        "gcp_bucket = \"keras-examples\"\n",
        "\n",
        "checkpoint_path = os.path.join(\"gs://\", gcp_bucket, \"mnist_example\", \"save_at_{epoch}\")\n",
        "\n",
        "tensorboard_path = os.path.join(  # Timestamp included to enable timeseries graphs\n",
        "    \"gs://\", gcp_bucket, \"logs\", datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
        ")\n",
        "\n",
        "callbacks = [\n",
        "    # TensorBoard will store logs for each epoch and graph performance for us.\n",
        "    keras.callbacks.TensorBoard(log_dir=tensorboard_path, histogram_freq=1),\n",
        "    # ModelCheckpoint will save models after each epoch for retrieval later.\n",
        "    keras.callbacks.ModelCheckpoint(checkpoint_path),\n",
        "    # EarlyStopping will terminate training when val_loss ceases to improve.\n",
        "    keras.callbacks.EarlyStopping(monitor=\"val_loss\", patience=3),\n",
        "]\n",
        "\n",
        "model = create_model()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "45d6210176e6"
      },
      "source": [
        "Here, we will load our data from Keras directly. In general, it's best practice\n",
        "to store your dataset in your Cloud Storage bucket, however TensorFlow Cloud can\n",
        "also accomodate datasets stored locally. That's covered in the Multi-file section\n",
        "of this guide."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bd4ef6ffa611"
      },
      "outputs": [],
      "source": [
        "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b1d2a2688887"
      },
      "source": [
        "The [TensorFlow Cloud](https://github.com/tensorflow/cloud) API provides the\n",
        "`remote()` function to determine whether code is being executed locally or on\n",
        "the cloud. This allows for the separate designation of `fit()` parameters for\n",
        "local and remote execution, and provides means for easy debugging without overloading\n",
        "your local machine."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cfab9ff41fd5"
      },
      "outputs": [],
      "source": [
        "if tfc.remote():\n",
        "    epochs = 100\n",
        "    callbacks = callbacks\n",
        "    batch_size = 128\n",
        "else:\n",
        "    epochs = 5\n",
        "    batch_size = 64\n",
        "    callbacks = None\n",
        "\n",
        "model.fit(x_train, y_train, epochs=epochs, callbacks=callbacks, batch_size=batch_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9b27c0b3b7db"
      },
      "source": [
        "Let's save the model in GCS after the training is complete."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b00451dcfeab"
      },
      "outputs": [],
      "source": [
        "save_path = os.path.join(\"gs://\", gcp_bucket, \"mnist_example\")\n",
        "\n",
        "if tfc.remote():\n",
        "    model.save(save_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0dceb5b7a173"
      },
      "source": [
        "We can also use this storage bucket for Docker image building, instead of your local\n",
        "Docker instance. For this, just add your bucket to the `docker_image_bucket_name` parameter."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "13200523ed93"
      },
      "outputs": [],
      "source": [
        "# docs_infra: no_execute\n",
        "tfc.run(docker_image_bucket_name=gcp_bucket)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "060a2112c34e"
      },
      "source": [
        "After training the model, we can load the saved model and view our TensorBoard logs\n",
        "to monitor performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b8d773e2cfb7"
      },
      "outputs": [],
      "source": [
        "# docs_infra: no_execute\n",
        "model = keras.models.load_model(save_path)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "05d1d68bae5a"
      },
      "outputs": [],
      "source": [
        "!#docs_infra: no_execute\n",
        "!tensorboard dev upload --logdir \"gs://keras-examples-jonah/logs/fit\" --name \"Guide MNIST\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3785ece03a8f"
      },
      "source": [
        "## Large-scale projects\n",
        "\n",
        "In many cases, your project containing a Keras model may encompass more than one\n",
        "Python script, or may involve external data or specific dependencies. TensorFlow\n",
        "Cloud is entirely flexible for large-scale deployment, and provides a number of\n",
        "intelligent functionalities to aid your projects.\n",
        "\n",
        "### Entry points: support for Python scripts and Jupyter notebooks\n",
        "\n",
        "Your call to the `run()` API won't always be contained inside the same Python script\n",
        "as your model training code. For this purpose, we provide an `entry_point` parameter.\n",
        "The `entry_point` parameter can be used to specify the Python script or notebook in\n",
        "which your model training code lives. When calling `run()` from the same script as\n",
        "your model, use the `entry_point` default of `None`.\n",
        "\n",
        "### `pip` dependencies\n",
        "\n",
        "If your project calls on additional `pip` dependencies, it's possible to specify\n",
        "the additional required libraries by including a `requirements.txt` file. In this\n",
        "file, simply put a list of all the required dependencies and TensorFlow Cloud will\n",
        "handle integrating these into your cloud build.\n",
        "\n",
        "### Python notebooks\n",
        "\n",
        "TensorFlow Cloud is also runnable from Python notebooks. Additionally, your specified\n",
        "`entry_point` can be a notebook if needed. There are two key differences to keep\n",
        "in mind between TensorFlow Cloud on notebooks compared to scripts:\n",
        "\n",
        "- When calling `run()` from within a notebook, a Cloud Storage bucket must be specified\n",
        "for building and storing your Docker image.\n",
        "- GCloud authentication happens entirely through your authentication key, without\n",
        "project specification. An example workflow using TensorFlow Cloud from a notebook\n",
        "is provided in the \"Putting it all together\" section of this guide.\n",
        "\n",
        "### Multi-file projects\n",
        "\n",
        "If your model depends on additional files, you only need to ensure that these files\n",
        "live in the same directory (or subdirectory) of the specified entry point. Every file\n",
        "that is stored in the same directory as the specified `entry_point` will be included\n",
        "in the Docker image, as well as any files stored in subdirectories adjacent to the\n",
        "`entry_point`. This is also true for dependencies you may need which can't be acquired\n",
        "through `pip`\n",
        "\n",
        "For an example of a custom entry-point and multi-file project with additional pip\n",
        "dependencies, take a look at this multi-file example on the\n",
        "[TensorFlow Cloud Repository](https://github.com/tensorflow/cloud/tree/master/src/python/tensorflow_cloud/core/tests/examples/multi_file_example).\n",
        "For brevity, we'll just include the example's `run()` call:\n",
        "\n",
        "```python\n",
        "tfc.run(\n",
        "    docker_image_bucket_name=gcp_bucket,\n",
        "    entry_point=\"train_model.py\",\n",
        "    requirements=\"requirements.txt\"\n",
        ")\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "997e3f89c734"
      },
      "source": [
        "## Machine configuration and distributed training\n",
        "\n",
        "Model training may require a wide range of different resources, depending on the\n",
        "size of the model or the dataset. When accounting for configurations with multiple\n",
        "GPUs, it becomes critical to choose a fitting\n",
        "[distribution strategy](https://www.tensorflow.org/guide/distributed_training).\n",
        "Here, we outline a few possible configurations:\n",
        "\n",
        "### Multi-worker distribution\n",
        "Here, we can use `COMMON_MACHINE_CONFIGS` to designate 1 chief CPU and 4 worker GPUs.\n",
        "\n",
        "```python\n",
        "tfc.run(\n",
        "    docker_image_bucket_name=gcp_bucket,\n",
        "    chief_config=tfc.COMMON_MACHINE_CONFIGS['CPU'],\n",
        "    worker_count=2,\n",
        "    worker_config=tfc.COMMON_MACHINE_CONFIGS['T4_4X']\n",
        ")\n",
        "```\n",
        "By default, TensorFlow Cloud chooses the best distribution strategy for your machine\n",
        "configuration with a simple formula using the `chief_config`, `worker_config` and\n",
        "`worker_count` parameters provided.\n",
        "\n",
        "- If the number of GPUs specified is greater than zero, `tf.distribute.MirroredStrategy` will be chosen.\n",
        "- If the number of workers is greater than zero, `tf.distribute.experimental.MultiWorkerMirroredStrategy` or `tf.distribute.experimental.TPUStrategy` will be chosen based on the accelerator type.\n",
        "- Otherwise, `tf.distribute.OneDeviceStrategy` will be chosen."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e0d938efab72"
      },
      "source": [
        "### TPU distribution\n",
        "Let's train the same model on TPU, as shown:\n",
        "```python\n",
        "tfc.run(\n",
        "    docker_image_bucket_name=gcp_bucket,\n",
        "    chief_config=tfc.COMMON_MACHINE_CONFIGS[\"CPU\"],\n",
        "    worker_count=1,\n",
        "    worker_config=tfc.COMMON_MACHINE_CONFIGS[\"TPU\"]\n",
        ")\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d1dec83a0b19"
      },
      "source": [
        "### Custom distribution strategy\n",
        "To specify a custom distribution strategy, format your code normally as you would\n",
        "according to the\n",
        "[distributed training guide](https://www.tensorflow.org/guide/distributed_training)\n",
        "and set `distribution_strategy` to `None`. Below, we'll specify our own distribution\n",
        "strategy for the same MNIST model.\n",
        "```python\n",
        "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
        "\n",
        "mirrored_strategy = tf.distribute.MirroredStrategy()\n",
        "with mirrored_strategy.scope():\n",
        "  model = create_model()\n",
        "\n",
        "if tfc.remote():\n",
        "    epochs = 100\n",
        "    batch_size = 128\n",
        "else:\n",
        "    epochs = 10\n",
        "    batch_size = 64\n",
        "    callbacks = None\n",
        "\n",
        "model.fit(\n",
        "    x_train, y_train, epochs=epochs, callbacks=callbacks, batch_size=batch_size\n",
        ")\n",
        "\n",
        "tfc.run(\n",
        "    docker_image_bucket_name=gcp_bucket,\n",
        "    chief_config=tfc.COMMON_MACHINE_CONFIGS['CPU'],\n",
        "    worker_count=2,\n",
        "    worker_config=tfc.COMMON_MACHINE_CONFIGS['T4_4X'],\n",
        "    distribution_strategy=None\n",
        ")\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0a50b62bf672"
      },
      "source": [
        "## Custom Docker images\n",
        "\n",
        "By default, TensorFlow Cloud uses a\n",
        "[Docker base image](https://hub.docker.com/r/tensorflow/tensorflow/)\n",
        "supplied by Google and corresponding to your current TensorFlow version. However,\n",
        "you can also specify a custom Docker image to fit your build requirements, if necessary.\n",
        "For this example, we will specify the Docker image from an older version of TensorFlow:\n",
        "```python\n",
        "tfc.run(\n",
        "    docker_image_bucket_name=gcp_bucket,\n",
        "    base_docker_image=\"tensorflow/tensorflow:2.1.0-gpu\"\n",
        ")\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bb659015ffad"
      },
      "source": [
        "## Additional metrics\n",
        "\n",
        "You may find it useful to tag your Cloud jobs with specific labels, or to stream\n",
        "your model's logs during Cloud training.\n",
        "It's good practice to maintain proper labeling on all Cloud jobs, for record-keeping.\n",
        "For this purpose, `run()` accepts a dictionary of labels up to 64 key-value pairs,\n",
        "which are visible from the Cloud build logs. Logs such as epoch performance and model\n",
        "saving internals can be accessed using the link provided by executing `tfc.run` or\n",
        "printed to your local terminal using the `stream_logs` flag.\n",
        "```python\n",
        "job_labels = {\"job\": \"mnist-example\", \"team\": \"keras-io\", \"user\": \"jonah\"}\n",
        "\n",
        "tfc.run(\n",
        "    docker_image_bucket_name=gcp_bucket,\n",
        "    job_labels=job_labels,\n",
        "    stream_logs=True\n",
        ")\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b34a2e8e09c3"
      },
      "source": [
        "## Putting it all together\n",
        "\n",
        "For an in-depth Colab which uses many of the features described in this guide,\n",
        "follow along\n",
        "[this example](https://github.com/tensorflow/cloud/blob/master/src/python/tensorflow_cloud/core/tests/examples/dogs_classification.ipynb)\n",
        "to train a state-of-the-art model to recognize dog breeds from photos using feature\n",
        "extraction."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "training_keras_models_on_cloud.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
